feat: Support logprobs for vLLM models in OpenAI Frontend#8538
Merged
pskiran1 merged 23 commits intoDec 4, 2025
Merged
Conversation
…probs-in-triton-openai' of https://github.com/triton-inference-server/server into spolisetty/tri-216-add-support-for-logprobs-and-top_logprobs-in-triton-openai
…and-top_logprobs-in-triton-openai
…probs-in-triton-openai' of https://github.com/triton-inference-server/server into spolisetty/tri-216-add-support-for-logprobs-and-top_logprobs-in-triton-openai
pskiran1
changed the title
logprobs for vLLM models in OpenAI APIlogprobs for vLLM models in OpenAI Frontend
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for logprobs (log probabilities) functionality in the OpenAI-compatible frontend for vLLM models. The feature allows users to request detailed probability information for generated tokens, which is useful for understanding model confidence and exploring alternative completions.
Key changes:
- Added logprobs support for both chat completions and standard completions endpoints
- Implemented conversion from vLLM's logprobs format to OpenAI's format
- Added comprehensive test coverage for logprobs functionality including validation and streaming
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| python/openai/openai_frontend/engine/utils/triton.py | Added helper functions to parse and convert logprobs from vLLM responses to OpenAI format for both chat and completion endpoints |
| python/openai/openai_frontend/engine/triton_engine.py | Integrated logprobs support into request handling, validation, and response generation for both streaming and non-streaming modes |
| python/openai/tests/test_openai_client.py | Added async tests for logprobs functionality using the OpenAI client library, including validation tests |
| python/openai/tests/test_chat_completions.py | Added HTTP-level tests for chat completions with logprobs, including edge cases and validation |
| python/openai/tests/test_completions.py | Added HTTP-level tests for completions with logprobs, including edge cases and validation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
pskiran1
marked this pull request as ready for review
November 24, 2025 12:48
whoisj
reviewed
Nov 24, 2025
yinggeh
reviewed
Nov 26, 2025
pskiran1
deleted the
spolisetty/tri-216-add-support-for-logprobs-and-top_logprobs-in-triton-openai
branch
December 4, 2025 04:37
20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does the PR do?
This PR adds support for
logprobsfunctionality in the OpenAI-compatible frontend for vLLM models. The feature allows users to request detailed probability information for generated tokens, which is useful for understanding model confidence and exploring alternative completions.Key changes:
Background:
https://platform.openai.com/docs/api-reference/completions/create
https://platform.openai.com/docs/api-reference/chat/create
Checklist
<commit_type>: <Title>Commit Type:
Check the conventional commit type
box here and add the label to the github PR.
Related PRs:
Where should the reviewer start?
Test plan:
Caveats:
Background
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)